Methods and apparatus for capturing images of an environment

ABSTRACT

Customer wide angle lenses and methods and apparatus for using such lenses in individual cameras as well as pairs of cameras intended for stereoscopic image capture are described. The lenses are used in combination with sensors to capture different portions of an environment at different resolutions. In some embodiments ground is capture at a lower resolution than sky which is captured at a lower resolution than a horizontal area of interest. Various asymmetries in lenses and/or lens and sensor placement are described which are particularly well suited for stereoscopic camera pairs where the proximity of one camera to the adjacent camera may interfere with the field of view of the cameras.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/239,823 filed Aug. 17, 2016 which published as U.S. patentpublication US 2018-0054607 A1 on Feb. 22, 2018, said patent applicationand patent publication of the application being hereby expresslyincorporated by reference their entirety.

FIELD

The present application relates to methods and apparatus for use incapturing images of an environment, e.g., images which can be used by aplayback device to simulate an environment in a non-stereoscopic orstereoscopic format.

BACKGROUND

Stereoscopic image capture of real world environments is of growingimportance as 3D viewing devices, e.g., head mounted displays, capableof displaying different images to a users left and right eyes arebecoming more common. For example, the capture of images of a sportingevent or other event such as a concert using cameras and thentransmitting the images to be used as textures on the surfaces of asimulated environment can give a user of a playback device a sense ofbeing at the event.

To capture images of an event for such purposes wide angle lenses, suchas a circular fisheye lenses, are often used. FIG. 1 illustrates theshape of a circular fish eye lens 100 as seen from the front. Thecircular fish eye lens is common and frequently used to capture imagesof an environment. Such a lens has a shape which approximates that of ahalf sphere. Circular fisheye lenses normally take in a 180° hemisphereand project this as a circle on the camera sensor such as the sensor 200shown in FIG. 2. Camera sensors are normally rectangular with one sidebeing longer than the other.

Circular fish eye lenses allow for the capture of a very wide angle,e.g., 180 or approximately 180 degrees in both the horizontal andvertical dimensions. This can be particularly useful when capturingimages which are to be used in simulating a complete 360 degreeenvironment in both vertical and horizontal directions since a full ornearly full view can be captured with as little as two lenses.

Unfortunately, the use of circular fish eye lenses have the drawbackthat they are inefficient in terms of available pixel sensor elementswith much of the sensor area including pixel sensors being wasted. FIG.2 shows how light 101 passing through the fish eye lens 100 willtypically fall on a sensor 200 of a camera device when the fish eye lens100 is used. Note that the pixel sensor outside the region 101 in whichlight falls are wasted leaving many of the pixel elements of sensor 200unused.

In an attempt to address the failure of circular fish eye lenses to usethe full area of a sensor, fisheye lenses that enlarged the image circleto cover the entire sensor area were developed with such lensessometimes being referred to as full-frame fisheye lenses.

The picture angle produced by conventional full-frame fish eye lensesonly measures 180 degrees when measured from corner to corner. Suchlenses have a 180° diagonal angle of view, while the horizontal andvertical angles of view will be smaller. For an equisolid angle-type 15mm full-frame fisheye, the horizontal FOV will be 147°, and the verticalFOV will be 94°

Such full-frame fisheye lenses treat the top and bottom portions of theenvironment equally with the same number of pixel elements of the sensorbeing allocated to the top horizontal portion of a FOV as to the thebottom horizontal portion of a field of view. It should be appreciatedthat conventional full-frame fish eye lenses tend to be symmetric withrespect to the left and right portions of the field of view as well asthe top and bottom portions of the field of view in the same manner thatcircular fish eye lenses are symmetric in the horizontal and verticaldirections.

While conventional circular fisheye lenses and conventional full framefisheye lenses have a wide range of applications, they are not as wellsuited to the capture of images of real world environments for virtualreality applications as many people would desire.

As should be appreciated, users of a virtual reality device tend not togive the same importance to all portions of an environment. For example,the ground may be of little interest to a user of a virtual realitydevice. In the case of the capture of images for virtual reality, bothnon-stereoscopic and/or stereoscopic virtual reality, it would bedesirable if different portions of the environment could be captured ina manner that uses sensor pixel elements in a way that reflects therelative importance of various portions of the environment to a user ofa virtual reality device, e.g., with higher priority portions of anenvironment being captured at higher resolution than lower priorityportions.

Some virtual reality applications do not involve the use of stereoscopicimages or use stereoscopic image pairs generated synthetically fromimages captured by multiple cameras and depth information. However realtime 3D image content capture and streaming often involves the use ofcamera pairs to capture images which are intended to serve as left andright eye images. Slight differences in the left and right images of astereoscopic image pair provide depth information allowing a userviewing different left and right images to perceive the images in 3D.Such capture of stereoscopic image pairs can avoid the need formanipulating an image to synthetically generate a corresponding image tobe used as one image of a stereoscopic image pair. Such processing canbe computationally difficult and time consuming making such processingunsuitable for some real time applications.

While not important for all applications, it would be desirable if atleast some consideration were given in the development and use of wideangle lenses to how they operate in a stereoscopic pair and methodsand/or apparatus be developed which facilitate the use of wide anglelenses in a stereoscopic camera pair.

In view of the above discussion, it should be appreciated that there isa need for improved methods and/or apparatus relating to wide anglelenses or devices which use such lenses in capturing images intended tosupport virtual reality applications whether such applications arenon-stereoscopic or stereoscopic applications.

SUMMARY

Methods and apparatus for capturing images which are well suited for usein environmental simulations are described. The methods and apparatuscan be used to capture stereoscopic or non-stereoscopic images which canbe processed and transmitted to a playback device for use in anenvironmental simulation, e.g., to give a user of a playback device asense of being present at an event such as a sports game or concertwhere the images are captured.

In various embodiments, wide angle lenses, e.g., custom fish eye lenses,which make efficient use of a camera sensors limited pixel sensors, areused. The custom fish eye lenses have a larger vertical and/orhorizontal field of view (FOV) than standard full-frame fisheye lenseswith a horizontal FOV of 147°, and a vertical FOV of 94°.

In various embodiments the vertical FOV of the custom fish eye lenses isat least 120 degrees but in many embodiments 160 degrees or more withthe FOV being at least 180 degrees in some embodiments. In some but notnecessarily all embodiments the horizontal FOV of the custom fish eyelenses is greater than 120 degrees but in many cases is in excess of 150degrees and in some embodiments over 160 degrees. The horizontal FOV is180 degrees in some embodiments.

In various embodiments the custom fish eye lens is used in combinationwith a sensor to capture images with different resolutions for differentportions of the environment. Portions of the sky and ground are oftencaptured using the fish eye lenses at lower resolution than a horizontalband corresponding to an area that may be described as the horizon inthe field of view of the lens.

In some but not all embodiments the lens is configured so that images ofsky portions are captured at lower resolution than horizon portions butwith ground portions being captured at even a lower resolution thaneither the sky or horizon portions of the environment. Such differencesin image capture may be, and sometimes are, due to vertical asymmetriesin a lens used to capture an image or images of the environment.

Various horizontal asymmetries are used in some embodiments in whichfish eye lenses are used in left and right cameras of a camera pair usedfor stereoscopic image capture. In various embodiments horizontal lensasymmetries are used to avoid or minimize a left lens of a camera paircapturing an image portion including the right camera or right cameralens. Similarly a right lens of a camera pair may include a horizontalasymmetry designed to minimize capturing of an image of the right cameraor right camera lens. In at least one embodiment the right camera usesan asymmetric lens which is the same as the one used in the left camerabut with the lens mounted in an inverted manner in the right camera ascompared to the left camera.

While in some embodiments the horizontal asymmetry between left andright camera image capture is achieved by using asymmetric cameralenses, in other embodiments the placement of the left camera lensrelative to the sensor of the left camera is shifted as compared to theplacement of the right camera lens to the sensor of the right camera. Inthis way, portions of light passing through the right and left cameralenses which correspond to a portion of the adjacent camera of thecamera pair will miss the sensor of the camera in which the lens ismounted allowing the individual pixel elements sensors of each camerasensor to be used to capture meaningful portions of the environmentrather than images of the adjacent camera or lens.

Various features of the present invention can be used alone or incombination and the described combinations are intended to be exemplaryand not limiting. For example, while various features are described asbeing used to capture stereoscopic image content using pairs of cameras,the methods and apparatus can be used with single cameras to capturenon-stereoscopic images of an environment.

Numerous additional features benefits and embodiments are described inthe detailed description which follows.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a circular fish eye lens.

FIG. 2 illustrates how a circular fish eye lens directs light on to arectangular camera sensor wasting large portions of the sensor.

FIG. 3 illustrates custom wide angle lenses, e.g., left and rightnon-circular fish eye lens.

FIG. 4 is a diagram illustrating how custom wide angle lens of the typeshown in FIG. 3 and having a first shape capture light from anenvironment including a sky, ground and horizon portion and direct thelight onto the majority of rectangular sensors of left and right camerasof a stereoscopic camera pair in accordance with one embodiment.

FIG. 5 is a diagram illustrating how custom wide angle lens of the typeshown in FIG. 3 and having a second shape capture light from anenvironment including a sky, ground and horizon portion and direct thelight onto the majority of rectangular sensors of left and right camerasof a stereoscopic camera pair in accordance with one embodiment.

FIG. 6 is a diagram illustrating left and right wide angle lensessimilar to the ones shown in FIG. 3 but which are longer than the onesin FIG. 3 in the horizontal dimension such that a portion of the lightcaptured from the environment using the lenses shown in FIG. 6 will missthe rectangular image sensors of left and right cameras in which thelenses of FIG. 6 are mounted and used.

FIG. 7 illustrates how custom wide angle lens of the type shown in FIG.6 and having a first shape capture light from an environment including asky, ground and horizon portion and direct the light onto the majorityof the rectangular sensors corresponding to left and right cameras of astereoscopic camera pair in accordance with one embodiment.

FIG. 8 illustrates how custom wide angle lens of the type shown in FIG.6 and having a second shape capture light from the environment includinga sky, ground and horizon portion and direct the light onto the majorityof the rectangular sensors corresponding to the left and right camerasof a stereoscopic camera pair in accordance with another embodiment.

FIG. 9 illustrates a stereoscopic camera pair and how the field of viewof a conventional circular fish eye lens may capture an image of aportion of an adjacent camera lens absent careful placement of thecamera sensors in accordance with one feature of the invention or use ofan asymmetric lens in accordance with another features of the invention.

FIG. 10 illustrates how the piece elements of sensors of a stereoscopiccamera pair can be efficiently used when asymmetric fish eye lenses inaccordance with the invention are used to capture images of anenvironment.

FIG. 11 illustrates an exemplary system implemented in accordance withsome embodiments of the invention which can be used to capture content,stream content, and output content to one or more users playback devicesin accordance with any of the embodiments described herein.

FIG. 12 illustrates a camera rig including multiple camera pairs forcapturing left and right eye images corresponding to different sectorsof a 360 degree field of view along with a camera or cameras directedtowards the sky to capture a sky view.

FIG. 13 illustrates an exemplary content delivery system with encodingcapability that can be used to encode and stream content in accordancewith the features of the invention.

DETAILED DESCRIPTION

Methods and apparatus are directed to wide angle lenses and apparatususing wide angle lenses which are well suited for use in capturingimages for virtual reality applications. In some embodiments fish eyelenses optimized for capture of images intended for virtual realityapplications are used to capture images of an environment. In at leastsome such embodiments the lenses capture different portions of theenvironment at different resolutions.

Drawing 300 of FIG. 3 illustrates exemplary custom wide angle lenses,e.g., left and right non-circular fish eye lens, in accordance with anexemplary embodiment. A first custom wide angle lens pair of the typeshown in FIG. 3, having a first shape is represented by lens pair (302,304). A second custom wide angle lens pair of the type shown in FIG. 3,having a second shape is represented by lens pair (302′, 304′).

In one embodiment sky and ground portions are intentionally captured atlower resolution by lenses (302, 304) and (302′, 304′) than portions ofthe environment corresponding to the horizon. Furthermore, unlikeconventional full frame fish eye lenses with vertical fields of view of147 degrees, the customized fish eye lenses (302, 304) and (302′, 304′)have a relatively large vertical field of view, e.g. 160 degrees or morein some embodiments. In some embodiments the horizontal field of view isat least 120 degrees but in many embodiments larger, e.g. in many caseswith a 140 degree FOV or larger. In some embodiments both the horizontaland vertical fields of view exceed 160 degrees.

FIG. 4 is a drawing 400 illustrating how exemplary custom wide anglelens, e.g., lens (302, 304) of the type shown in FIG. 3 and having afirst shape capture light from an environment including a sky, groundand horizon portion and direct the light onto the majority ofrectangular sensors (402, 404) of left and right cameras of astereoscopic camera pair in accordance with one embodiment.

Sensor 402 includes sensor portions (406, 408, 410, 412, 412′, 412″,412′″). Sensor portion 406 of sensor 402 captures light from a skyportion of the environment which was directed by lens 302, onto sensorportion 406. Sensor portion 408 of sensor 402 captures light from a mainviewing area portion, e.g. horizontal viewing portion, of theenvironment directed by lens 302 onto sensor portion 408. Sensor portion410 of sensor 402 captures light from a ground portion of theenvironment which was directed by lens 302 onto sensor portion 410.Sensor portions (412, 412′ 412″, 412′″) are wasted portions of therectangular sensor 402. Portion 409 included as part of portion 408captures a portion of lens 304.

Sensor 404 includes sensor portions (416, 418, 420, 422, 422′, 422″,422′″). Sensor portion 416 of sensor 404 captures light from a skyportion of the environment which was directed by lens 304, onto sensorportion 416. Sensor portion 418 of sensor 404 captures light from a mainviewing area portion, e.g. horizontal viewing portion, of theenvironment directed by lens 304 onto sensor portion 418. Sensor portion420 of sensor 404 captures light from a ground portion of theenvironment which was directed by lens 304 onto sensor portion 420.Sensor portions (422, 422′ 422″, 422′″) are wasted portions of therectangular sensor 404. Portion 419 included as part of portion 418captures a portion of lens 302.

Lenses 302 and 304 are symmetric in their field of view (FOV) in thehorizontal and vertical directions, Accordingly lenses 302 and 304 areof the same type and can be used interchangeably.

Unlike the first shape lenses (302, 304) the second shape lenses (302′,304′) are asymmetric with respect to the vertical dimension allocatingmore sensor area for the sky portion and less than the ground portioneven though with respect to the horizon these portions maybe of equalsize. Thus, in lenses 302′ and 304′ are symmetric with respect to thevertical axis but not the horizontal axis. Such a user or pixel elementsof a sensor can be more efficient since the ground portion is generallyof less interest than the sky portions so in allocating a limited numberof pixel sensors it is desirable to minimize the number allocated to theground portion of the environment, maximize the allocation of pixelsensors and thus resolution of the horizon portion and capture the skyat a lower resolution than the horizon portion but higher than theground portion.

Thus, in some embodiments the custom fish eye lenses (302′, 304′) areasymmetric with regard to the resolution allocation provided for thevertical field of view and/or are placed in an asymmetric manner oversensors so that the images captured by cameras using the custom fish eyelens (302′, 304′) will capture a vertical portion of an environment inan asymmetric manner in terms of FOV and/or the resolution of thecaptured portions of the environment.

In one such embodiment while the vertical FOV may cover a number ofdegrees which is the same or similar for the sky and ground portions ofan environment, but with the sky portion being directed to more pixelelements of the sensor allowing the sky to be captured at a higherresolution than the ground portion. Thus asymmetry in terms ofallocation of resolution is particularly useful in the verticaldimension where objects in the sky such a planes, birds, clouds are morelikely to be of interest than objects on the ground.

FIG. 5 is a drawing 500 illustrating how custom wide angle lens (302′,304′) of the type shown in FIG. 3 and having a second shape capturelight from an environment including a sky, ground and horizon portionand direct the light onto the majority of rectangular sensors (502, 504)of left and right cameras of a stereoscopic camera pair in accordancewith one embodiment where the sensors are asymmetric with regard to thevertical direction but not the horizontal direction.

Sensor 502 includes sensor portions (506, 508, 510, 512, 512′, 512″,512′″). Sensor portion 506 of sensor 502 captures light from a skyportion of the environment which was directed by lens 302′, onto sensorportion 506. Sensor portion 508 of sensor 502 captures light from a mainviewing area portion, e.g. horizontal viewing portion, of theenvironment directed by lens 302′ onto sensor portion 508 at a higherresolution than the sky or ground portions. Sensor portion 510 of sensor502 captures light from a ground portion of the environment which wasdirected by lens 302′ onto sensor portion 510. Sensor portion 510 issmaller than the sensor area 506 corresponding to the sky portion andcaptures the ground at a lower resolution than the resolution at whichthe sky and main horizontal viewing area are captured. Sensor portions(512, 512′ 512″, 512′″) are wasted portions of the rectangular sensor502. Portion 509 included as part of portion 508 captures a portion oflens 304′.

Sensor 504 includes sensor portions (516, 518, 520, 522, 522′, 522″,522′″). Sensor portion 516 of sensor 504 captures light from a skyportion of the environment which was directed by lens 304′ which isasymmetric in the vertical direction, onto sensor portion 516. Sensorportion 518 of sensor 504 captures light from a main viewing areaportion, e.g. horizontal viewing portion, of the environment directed bylens 304′ onto sensor portion 518. Sensor portion 520 of sensor 504captures light from a ground portion of the environment which wasdirected by lens 304′ onto sensor portion 520. Sensor portions (522,422′ 422″, 422′″) are wasted portions of the rectangular sensor 504.Portion 519 included as part of portion 518 captures a portion of lens302′. As with sensor 502, the ground is captured at a lower resolutionthan the sky which is captured at a lower resolution than the mainhorizontal viewing area.

The FOV of lenses 302′, 304′ maybe the same as lenses 302, 304 but byusing lower resolution for the ground more efficient use of the pixelelements of sensor 502, 505 better image quality can be achieved for thehigher priority portions of the environment when images are capturedusing the lenses 302′, 304′ on a camera pair than when lenses 302, 304are used.

Since lenses 302′, 304′ are symmetric in the horizontal dimension, thesame type of lens can be used for either the left or right cameras of astereoscopic camera pair with the lenses 302′, 304′ beinginterchangeable.

Implementations which allow the same type of lenses for left and rightcameras of a stereoscopic camera pair can have cost advantages overimplementations which involve the use of lenses of different designs forthe left and right cameras of a stereo camera pair.

However, asymmetry in the captured FOV and/or resolution allocation inthe horizontal dimension is particularly useful when fish eye lenses areused in camera pairs, e.g., for stereoscopic image capture. In such acase the left camera of the camera pair may have its right side viewpartially obstructed by the right camera of the camera pair but its leftfield of view unobstructed. Similarly the right camera of the camerapair may have its left side view partially obstructed by the rightcamera of the camera pair but its right side view unobstructed. Suchobstruction results in a sensor capturing a portion 409, 419, 509 or 519of a lens of an adjacent camera as shown in FIGS. 4 and 5. The captureof a portion of the adjacent lens can be viewed as a waste of pixelsensors when the objective is to capture a quality image of theenvironment using a limited number of pixels sensors of a camera sensorsuch as the sensor 402, 404, 502 or 505.

The problem of capturing an image of a portion of an adjacent lens canbe understood with regard to FIG. 9.

In order to avoid wasting the limited number of pixel element sensors ona sensor capturing a view of the adjacent camera or lens of the adjacentcamera, custom fish eye lens with asymmetric fields of view can be, andsometimes are, used so that the adjacent camera or lens of the adjunctcamera is not captured. Given the symmetry between the left and rightcameras, in some embodiments the right camera uses an asymmetric lenswhich the same as that used in the left camera of the camera pair butwith the right lens being inverted as compared to the left lens. In sucha case, the left and right cameras use the same type asymmetric lens,but each camera uses the lens in an orientation which avoids orminimizes the capture of an image of a portion of the adjacent camera orlens assembly of the adjacent camera. In one such embodiment theresolution allocation for the sky and ground is the same, butpotentially lower than the resolution allocation to the horizon portionof the environment to allow for inverting of the lens in one of the leftand right cameras of a stereoscopic camera pair without creating aninconsistency in the resolution allocations between the left and rightcameras for portions of the environment.

In some embodiments, rather than use a fish eye lens which is asymmetricin the horizontal dimension for left and right camera pairs, theplacement of the left and right sensors is shifted in the left and rightcameras corresponding to the lens so that the FOV of the lens whichcaptures the adjacent camera or lens will particularly or fully avoidthe sensor of the camera. In this way the pixel sensors of an individualone of the left and right cameras can be utilized to capture portions ofthe environment that are of interest as opposed to an image of theadjacent camera or camera lens. The asymmetric placement of the customfish eye lens to the sensor in the left and right cameras of astereoscopic camera pair is useful in embodiments where the resolutionallocation to the sky and ground portions are different making itundesirable to use the same style lens in both left and right cameras ofa camera pair but with one of the lenses inverted relative to the other.

As should be appreciated by directing light from different size areasonto a fixed number of pixels for ground and sky luminance differenceswhich may not be present in the actual environment may be introducedinto the captured images. For example, if the light from a ground areaof one portion, e.g., a first percentage of the FOV, is directed onto afirst number of pixels, and the light from a sky portion of the samesize, e.g., same percentage of the FOV as the first percentage, isdirected, e.g., spread, over twice as many pixels, the ground area willappear excessively bright relative to the sky which is captured at ahigher resolution.

In various embodiments, the optical characteristics of the wide angle,e.g., custom fish eye lens, is determined, e.g., based on calibrationmeasurements, and the images captured by the fish eye lens are correctedto compensate for the luminance distortions introduced by the calibratedlens. Chrominance and other optical distortions can also be measured andcorrected for in the captured images. Thus, while the fish eye lenses ofthe invention may capture distorted images of the environment suchdistortions can be corrected for in post image capture processing. Thuswhile the lenses used in various embodiments may have variousasymmetries, post capture processing can be used to correct for thedistortions while allowing efficient use of the limited pixel elementsof a sensor in a manner that is optimized for the portions of theenvironment which are most likely to be of interest to a viewer of theimages or image portions during playback by a playback device displayinga simulated environment to a user.

The lenses. lens and sensor mounting arrangements, and correction ofdistortions introduced into the captured images are well suited for usewith devices which are intended to capture and display images of a realenvironment as part of a simulated environment displayed to a user of aplayback device whether the environmental simulation be a stereoscopicenvironmental simulation or non-stereoscopic environmental simulation

FIG. 9 illustrates a stereoscopic camera pair and how the field of viewof a fish eye lens, whether it be a conventional fish eye lens or customfish eye lens 302, 304, 302′ or 304′ of the invention may capture animage of a portion of an adjacent camera lens if the snesor is orienteddirectly under the center of the fish eye lens as is the normal case.

As will be discussed further below careful placement of the camerasensors in accordance with one feature of the invention or use of anasymmetric lens in accordance with another feature of the inventionallows the sensors to avoid capturing an image of a lens of an adjacentcamera of a stereoscopic camera pair. Drawing 900 illustrates a filed ofview for a stereoscopic camera pair (902, 904) each including a fish eyelens (906, 908), respectively. Camera 902 has a field of view within theboundaries of dashed lines 910 and 912. Camera 904 has a field of viewwithin the boundaries of dashed lines 914 and 916. It may be observedthat a portion of lens 908 of camera 904 is within the field of view forlens 906 of camera 902.

Similarly, it may be observed that a portion of lens 906 of camera 902is within the field of view for lens 908 of camera 904. Drawing 900indicates that sensor S1 920 is mounted in camera 902 to be aligned withthe lens collar 905 corresponding to lens 906. Drawing 900 furtherindicates that sensor S2 922 is mounted in camera 904 to be aligned withthe lens collar 907 corresponding to lens 908.

Drawing 950 illustrates that the sensor to lens placement shown in FIG.900 results in a wasted sensor area for each sensor due to the captureof undesired areas, e.g., including a portion of an adjacent lens.Sensor S1 920 includes a portion 923 which captures a desired area,represented by dotted shading, and a portion 924 which captures anundesired area, represented by crosshatch shading. Sensor S2 922includes a portion 925 which captures a desired area, represented bydotted shading, and a portion 926 which captures an undesired area,represented by crosshatch shading. For each sensor S1 920, S2 922, thewasted area (924, 926) may be more than 5% and in some cases as much as10-15% of the total sensor area (920, 922). However this is not the casein all systems and lower amounts of wastage may occur in some systems.

Drawing 975 illustrates a sensor placement adjustment solution, inaccordance with a feature of some embodiments of the present invention,in which the sensors (920, 920) are more fully utilized, e.g.,minimizing or eliminating wasted sensor area. Sensor S1 is 920 is movedin the direction opposite the side of lens which captures undesiredarea, as indicated by arrow 976. This adjustment or intentional offsetof sensor placement, results in entire or nearly entire sensor 1 920capturing desired area, as indicated by dotted shading over the regionof sensor S1 920. Sensor S1 is 920 is moved in the direction oppositethe side of lens which captures undesired area, as indicated by arrow978. This adjustment or intentional offset of sensor placement, resultsin entire or nearly entire sensor 2 922 capturing desired area, asindicated by dotted shading over the region of sensor S2 922. Thus thehorizontal center of lenses 920 and 922 are not aligned with thehorizontal centers of the lenses 906 and 908 as would be the case in anormal camera configuration.

FIGS. 6, 7 and 8 show how the offset alignment shown in FIG. 9 can beused with custom fisheye lenses having fields of view which are the sameor similar to those discussed with regard to lenses 302, 302′, 304,304′.

FIG. 6 is a diagram illustrating left and right wide angle lenses ((602or 602′), (604 or 604′)) similar to the ones ((302 or 302′), (304) or304′)) shown in FIG. 3 but which are longer than the ones ((302 or302′), (304) or 304′)) in FIG. 3 in the horizontal dimension such that aportion of the light captured from the environment using the lenses((602 or 602′), (604 or 604′)) shown in FIG. 6 will miss the rectangularimage sensors ((702 or 802), (704 or 804)) of left and right cameras inwhich the lenses ((602 or 602′), (604 or 604′)) of FIG. 6 are mountedand used.

Drawing 600 of FIG. 6 illustrates exemplary custom wide angle lenses,e.g., left and right non-circular fish eye lens, in accordance with anexemplary embodiment. A first custom wide angle lens pair of the typeshown in FIG. 6, having a first shape is represented by lens pair (602,604). A second custom wide angle lens pair of the type shown in FIG. 6,having a second shape is represented by lens pair (602′, 604′).

FIG. 7 is a drawing 700 illustrating how exemplary custom wide anglelens, e.g., lens (602, 604) of the type shown in FIG. 6 and having afirst shape capture light from an environment including a sky, groundand horizon portion and direct the light onto the majority ofrectangular sensors (702, 704) of left and right cameras of astereoscopic camera pair in accordance with one embodiment. Sensor 702includes sensor portions (706, 708, 710, 712, 712′, 712″, 712′″). Sensorportion 706 of sensor 702 captures light from a sky portion of theenvironment which was directed by lens 602, onto sensor portion 706.Sensor portion 708 of sensor 702 captures light from a main viewing areaportion, e.g. horizontal viewing portion, of the environment directed bylens 602 onto sensor portion 708. Sensor portion 710 of sensor 602captures light from a ground portion of the environment which wasdirected by lens 602 onto sensor portion 710. Sensor portions (712,712′, 712″, 712′″) are wasted portions of the rectangular sensor 702.Area 713 corresponds to a wasted portion of lens 602 for which there isnot a corresponding sensor portion. Area 714, included in area 713,corresponds to light captured from a portion of the viewing environmentof lens 602 which included a portion of lens 604, thus lens 604 is notcaptured by sensor 702.

Sensor 704 includes sensor portions (716, 718, 720, 722, 722′, 722″,722′″). Sensor portion 716 of sensor 704 captures light from a skyportion of the environment which was directed by lens 604, onto sensorportion 716. Sensor portion 718 of sensor 704 captures light from a mainviewing area portion, e.g. horizontal viewing portion, of theenvironment directed by lens 604 onto sensor portion 718. Sensor portion720 of sensor 704 captures light from a ground portion of theenvironment which was directed by lens 604 onto sensor portion 720.Sensor portions (722, 722′, 722″, 722′″) are wasted portions of therectangular sensor 704. Area 723 corresponds to a wasted portion of lens604 for which there is not a corresponding sensor portion. Area 724,included in area 723, corresponds to light captured from a portion ofthe viewing environment of lens 604 which included a portion of lens602, thus lens 602 is not captured by sensor 704.

FIG. 8 is a drawing 800 illustrating how exemplary custom wide anglelens, e.g., lens (602′, 604′) of the type shown in FIG. 6 and having asecond shape capture light from an environment including a sky, groundand horizon portion and direct the light onto the majority ofrectangular sensors (802, 804) of left and right cameras of astereoscopic camera pair in accordance with one embodiment. Sensor 802includes sensor portions (806, 808, 810, 812, 812′, 812″, 812′″). Sensorportion 806 of sensor 802 captures light from a sky portion of theenvironment which was directed by lens 602′, onto sensor portion 806.Sensor portion 808 of sensor 802 captures light from a main viewing areaportion, e.g. horizontal viewing portion, of the environment directed bylens 602′ onto sensor portion 808. Sensor portion 810 of sensor 802captures light from a ground portion of the environment which wasdirected by lens 602′ onto sensor portion 810. Sensor portions (812,812′, 812″, 812′″) are wasted portions of the rectangular sensor 802.Area 813 corresponds to a wasted portion of lens 602′ for which there isnot a corresponding sensor portion. Area 814, included in area 813,corresponds to light captured from a portion of the viewing environmentof lens 602′ which included a portion of lens 604′, thus lens 604′ isnot captured by sensor 802.

Sensor 804 includes sensor portions (816, 818, 820, 822, 822′, 822″,822′″). Sensor portion 816 of sensor 804 captures light from a skyportion of the environment which was directed by lens 604′, onto sensorportion 816. Sensor portion 818 of sensor 804 captures light from a mainviewing area portion, e.g. horizontal viewing portion, of theenvironment directed by lens 604′ onto sensor portion 818. Sensorportion 820 of sensor 704′ captures light from a ground portion of theenvironment which was directed by lens 604′ onto sensor portion 820.Sensor portions (822, 822′, 822″, 822′″) are wasted portions of therectangular sensor 804. Area 823 corresponds to a wasted portion of lens604′ for which there is not a corresponding sensor portion. Area 824,included in area 823, corresponds to light captured from a portion ofthe viewing environment of lens 604′ which included a portion of lens602′, thus lens 602′ is not captured by sensor 804.

Thus as can be appreciated from FIGS. 6, 7 and 8, by shifting theposition of the sensors used with the custom fish eye lenses of theinvention, capture of a portion of an adjacent camera lens can beavoided.

An alternative to using the sensor offset approach shown in FIGS. 7 and8 is to use a custom fish eye lens which is asymmetric in the horizontaldirection to avoid capture of an image of the lens of the adjacentcamera in a stereoscopic camera pair. Such an approach can and in someembodiments is combined with the approach of having an asymmetry in thevertical dimension as well to capture the ground at a lower resolutionthan the sky. Such an approach has the disadvantage that the lenses forthe left and right cameras of a stereoscopic camera pair need to bedifferent since the left lens is asymmetric in the horizontal dimensiondifferently than right lens assuming the lenses are also asymmetric inthe vertical direction. In the case where the sky and ground arecaptured at the same resolution the lenses maybe symmetric in thevertical direction and an inverted lens of the same type maybe used inthe right in the right camera of a camera pair while the left camerauses the lens in a non-inverted manner. The inversion shifts theasymmetry as needed to avoid capture of an image of an adjacent cameralens in such an embodiment.

FIG. 10 illustrates how the piece elements of sensors of a stereoscopiccamera pair can be efficiently used when asymmetric fish eye lenses inaccordance with the invention are used to capture images of aenvironment. Drawing 1000 of FIG. 10 illustrates steroo camera pair(1002, 1004) including custom non-symmetric wide angle lenses (1006,1008), lens collars (1005, 1007), and sensor (S1 1020, S2 1022),respectively. The field of view for lens 1006 of camera 1002 is a regionbetween dashed lines (1010, 1012). It may be observed that lens 1008 isoutside of the field of view of lens 1006 of camera 1002. The field ofview for lens 1008 of camera 1004 is a region between dashed lines(1014, 1016). It may be observed that lens 1006 is outside of the fieldof view of lens 1008 of camera 1004. In FIG. 10 the center of eachsensor is aligned with the center of the lens which is placed in frontof the sensor.

Drawing 1050 illustrates that the custom non-symmetric wide angles lens(1006, 1008) allow sensor area wastage to be minimized or eliminated.Sensor S1 1020 is aligned with lens collar 1005, and sensor S1 capturesdesired viewing area as indicated by dotted shading over the entiresensor S1 1020. Sensor S2 1022 is aligned with lens collar 1007, andsensor S2 1022 captures desired viewing area as indicated by dottedshading over the entire sensor S2 1022.

Each of the lenses described herein can be used on an individual cameraor camera of a camera pair. In the case of stereoscopic camera pairs,the pairs of lenses maybe used as discussed above. Thus while manyfeatures are well suited for use with steroscopic camera pairsindividual lens can be used in individual unpaired cameras. Accordingly,any of the above described lenses can and are, depending on theembodiment, used in a camera device such as the left camera deviceincluding a camera body 902, sensor 920 and, optionally, a lense collar905 as shown in FIG. 9. Thus the present application covers any camerasand camera pairs including the described lens configurations and/or lensand sensor placement configurations in a camera or camera system.

FIG. 11 illustrates an exemplary system 1100 implemented in accordancewith some embodiments of the invention. The system 1100 supports contentdelivery, e.g., imaging content delivery, to one or more customerdevices, e.g., playback devices/content players, located at customerpremises. The system 1100 includes the exemplary image capturing device102, a content delivery system 104, a communications network 105, and aplurality of customer premises 106, . . . , 110. The image capturingdevice 102 supports capturing of stereoscopic imagery. The imagecapturing device 102 captures and processes imaging content inaccordance with the features of the invention. The communicationsnetwork 105 may be, e.g., a hybrid fiber-coaxial (HFC) network,satellite network, and/or internet.

The content delivery system 104 includes an image processing,calibration and encoding apparatus 112 and a content delivery device,e.g. a streaming server 114. The image processing, calibration andencoding apparatus 112 is responsible for performing a variety offunctions including camera calibration based on one or more targetimages and/or grid patterns captured during a camera calibrationprocess. Content delivery device 114 may be implemented as a serverwith, as will be discussed below, the delivery device responding torequests for content with image calibration information, optionalenvironment information, and one or more images captured by the camerarig 102 which can be used in simulating a 3D environment. Streaming ofimages and/or content may be and sometimes is a function of feedbackinformation such as viewer head position and/or user selection of aposition at the event corresponding to a camera 102 which is to be thesource of the images. For example, a user may select or switch betweenimages from a camera rig positioned at center line to a camera rigpositioned at the field goal with the simulated 3D environment andstreamed images being changed to those corresponding to the userselected camera rig. Thus it should be appreciated that while a singlecamera rig 102 is shown in FIG. 1 multiple camera rigs may be present inthe system and located at different physical locations at a sporting orother event with the user being able to switch between the differentpositions and with the user selections being communicated from theplayback device 122 to the content server 114. While separate devices112, 114 are shown in the image processing and content delivery system104, it should be appreciated that the system may be implemented as asingle device including separate hardware for performing the variousfunctions or with different functions being controlled by differentsoftware or hardware modules but being implemented in or on a singleprocessor.

Encoding apparatus 112 may, and in some embodiments does, include one ora plurality of encoders for encoding image data in accordance with theinvention. The encoders may be used in parallel to encode differentportions of a scene and/or to encode a given portion of a scene togenerate encoded versions which have different data rates. Usingmultiple encoders in parallel can be particularly useful when real timeor near real time streaming is to be supported.

The content streaming device 114 is configured to stream, e.g.,transmit, encoded content for delivering the encoded image content toone or more customer devices, e.g., over the communications network 105.Via the network 105, the content delivery system 104 can send and/orexchange information with the devices located at the customer premises106, 110 as represented in the figure by the link 120 traversing thecommunications network 105.

While the encoding apparatus 112 and content delivery server are shownas separate physical devices in the FIG. 1 example, in some embodimentsthey are implemented as a single device which encodes and streamscontent. The encoding process may be a 3D, e.g., stereoscopic, imageencoding process where information corresponding to left and right eyeviews of a scene portion are encoded and included in the encoded imagedata so that 3D image viewing can be supported. The particular encodingmethod used is not critical to the present application and a wide rangeof encoders may be used as or to implement the encoding apparatus 112.

Each customer premise 106, 110 may include a plurality ofdevices/players, e.g., decoding apparatus to decode and playback/displaythe image content streamed by the content streaming device 114. Customerpremise 1 106 includes a decoding apparatus/playback device 122 coupledto a display device 124 while customer premise N 110 includes a decodingapparatus/playback device 126 coupled to a display device 128. In someembodiments the display devices 124, 128 are head mounted stereoscopicdisplay devices.

In various embodiments decoding apparatus 122, 126 present the imagingcontent on the corresponding display devices 124, 128. The decodingapparatus/players 122, 126 may be devices which are capable of decodingthe imaging content received from the content delivery system 104,generate imaging content using the decoded content and rendering theimaging content, e.g., 3D image content, on the display devices 124,128. Any of the decoding apparatus/playback devices 122, 126 may be usedas the decoding apparatus/playback device.

To facilitate an understanding of the image capture process referencewill now be made to the exemplary camera rig shown in FIG. 12. Thecamera rig 1300 can be used as the rig 102 of the FIG. 1 system andincludes a plurality of stereoscopic pairs each corresponding to adifferent one of three sectors. The camera of each camera pair includesa lens, sensor and camera body. The lens and sensors maybe of the typedescribed in any of the above discussed embodiments and the sensor tolens placement maybe as described in any of the above describedembodiments. The first camera pair 1301 includes a left eye camera 1302and a right camera 1304 intended to capture images corresponding tothose which would be seen by a left and right eye of a person. Secondsector camera pair 1305 includes left and right cameras 1306, 1308 whilethe third sector camera pair 1309 includes left and right cameras 1310,1312. Each camera is mounted in a fixed position in the supportstructure 1318. An upward facing camera 1314 is also included. Adownward facing camera which is not visible in FIG. 13 may be includedbelow camera 1314. Stereoscopic camera pairs are used in someembodiments to capture pairs of upward and downward images however inother embodiments a single upward camera and a single downward cameraare used. In still other embodiments a downward image is captured priorto rig placement and used as a still ground image for the duration of anevent. Such an approach tends to be satisfactory for many applicationsgiven that the ground view tends not to change significantly during anevent.

FIG. 13 illustrates an exemplary content delivery system 1700 withencoding capability that can be used to encode and stream content inaccordance with the features of the invention.

The system may be used to perform encoding, storage, and transmissionand/or content output in accordance with the features of the invention.The content delivery system 1700 may be used as the system 104 ofFIG. 1. While the system shown in FIG. 13 is used for encoding,processing and streaming of content, it should be appreciated that thesystem 1700 may also include the ability to decode and display processedand/or encoded image data, e.g., to an operator.

The system 1700 includes a display 1702, input device 1704, input/output(I/O) interface 1706, a processor 1708, network interface 1710 and amemory 1712. The various components of the system 1700 are coupledtogether via bus 1709 which allows for data to be communicated betweenthe components of the system 1700.

The memory 1712 includes various modules, e.g., routines, which whenexecuted by the processor 1708 control the system 1700 to implement thepartitioning, encoding, storage, and streaming/transmission and/oroutput operations in accordance with the invention.

The memory 1712 includes various modules, e.g., routines, which whenexecuted by the processor 1707 control the computer system 700 toimplement the immersive stereoscopic video acquisition, encoding,storage, and transmission and/or output methods in accordance with theinvention. The memory 1712 includes control routines 1714, apartitioning module 1716, encoder(s) 1718, a streaming controller 1720,received input images 1732, e.g., 360 degree stereoscopic video of ascene, encoded scene portions 1734, and timing information 1736. In someembodiments the modules are, implemented as software modules. In otherembodiments the modules are implemented in hardware, e.g., as individualcircuits with each module being implemented as a circuit for performingthe function to which the module corresponds. In still other embodimentsthe modules are implemented using a combination of software andhardware.

The control routines 1714 include device control routines andcommunications routines to control the operation of the system 1700. Thepartitioning module 1716 is configured to partition a receivedstereoscopic 360 degree version of a scene into N scene portions inaccordance with the features of the invention.

The encoder(s) 1718 may, and in some embodiments do, include a pluralityof encoders configured to encode received image content, e.g., 360degree version of a scene and/or one or more scene portions inaccordance with the features of the invention. In some embodimentsencoder(s) include multiple encoders with each encoder being configuredto encode a stereoscopic scene and/or partitioned scene portions tosupport a given bit rate stream. Thus in some embodiments each sceneportion can be encoded using multiple encoders to support multipledifferent bit rate streams for each scene. An output of the encoder(s)1718 is the encoded scene portions 1734 which are stored in the memoryfor streaming to customer devices, e.g., playback devices. The encodedcontent can be streamed to one or multiple different devices via thenetwork interface 1710.

UV maps 1740 are stored in memory 1712 of the content delivery system1700. The UV maps 1740 correspond to different resolution allocationsand/or areas of the environment. For example, the first UV map 1 1742corresponds to a first resolution allocation, the second UV map 2 1744corresponds to a second resolution allocation, and the third UV map 1746corresponds to a third resolution allocation. UV maps with differentresolution allocations can correspond to the same area of anenvironment. Different UV maps corresponding to other areas of theenvironment can be stored in the memory 1712. Multiple UV maps maycorrespond to the environmental model. The mesh model of the environmentwhere the received images were captured is stored in memory 1712 of thecontent delivery system 1700, e.g., 3D environmental mesh model 1738.Multiple mesh models may be stored in the memory 1712.

The streaming controller 1720 is configured to control streaming ofencoded content for delivering the encoded image content to one or morecustomer devices, e.g., over the communications network 105. Thestreaming controller 1720 includes a request processing module 1 722, adata rate determination module 1724, a current head positiondetermination module 1726, a selection module 1728 and a streamingcontrol module 1730. The request processing module 1722 is configured toprocess a received request for imaging content from a customer playbackdevice. The request for content is received in various embodiments via areceiver 1713 in the network interface 1710. In some embodiments therequest for content includes information indicating the identity ofrequesting playback device. In some embodiments the request for contentmay include data rates supported by the customer playback device, acurrent head position of the user, e.g., position of the head mounteddisplay. The request processing module 1722 processes the receivedrequest and provides retrieved information to other elements of thestreaming controller 1720 to take further actions. While the request forcontent may include data rate information and current head positioninformation, in various embodiments the data rate supported by theplayback device can be determined from network tests and other networkinformation exchange between the system 1700 and the playback device.

The data rate determination module 1724 is configured to determine theavailable data rates that can be used to stream imaging content tocustomer devices, e.g., since multiple encoded scene portions aresupported the content delivery system 1700 can support streaming contentat multiple data rates to the customer device. The data ratedetermination module 1724 is further configured to determine the datarate supported by a playback device requesting content from system 1700.In some embodiments the data rate determination module 1724 isconfigured to determine data rates for delivery of image content basedon network measurements.

The current head position determination module 1726 is configured todetermine a current viewing angle and/or a current head position of theuser, e.g., position of the head mounted display, from informationreceived from the playback device. In some embodiments the playbackdevice periodically sends current head position information to thesystem 1700 where the current head position determination module 1726receives and processes the information to determine the current viewingangle and/or a current head position.

The selection module 1728 is configured to determine which portions of a360 degree scene to stream to a playback device based on the currentviewing angle/head position information of the user. The selectionmodule 1728 is further configured to select the encoded versions of thedetermined scene portions based on the available data rates to supportstreaming of content.

The streaming control module 1730 is configured to control streaming ofimage content, e.g., multiple portions of a 360 degree stereoscopicscene, at various supported data rates in accordance with the featuresof the invention. In some embodiments the streaming control module 1730is configured to control the streaming of N portions of a 360 degreestereoscopic scene to the playback device requesting content toinitialize scene memory in the playback device. In various embodimentsthe streaming control module 1730 is configured to send the selectedencoded versions of the determined scene portions periodically, e.g., ata determined rate. In some embodiments the streaming control module 1730is further configured to send 360 degree scene updates to the playbackdevice in accordance with a time interval, e.g., once every minute. Insome embodiments sending 360 degree scene update includes sending Nscene portions or N-X scene portions of the full 360 degree stereoscopicscene, where N is the total number of portions into which the full 360degree stereoscopic scene has been partitioned and X represents theselected scene portions recently sent to the playback device. In someembodiments the streaming control module 1730 waits for a predeterminedtime after initially sending N scene portions for initialization beforesending the 360 degree scene update. In some embodiments the timinginformation to control sending of the 360 degree scene update isincluded in the timing information 1736. In some embodiments thestreaming control module 1730 is further configured identify sceneportions which have not been transmitted to the playback device during arefresh interval; and transmit an updated version of the identifiedscene portions which were not transmitted to the playback device duringthe refresh interval.

In various embodiments the streaming control module 1730 is configuredto communicate at least a sufficient number of the N portions to theplayback device on a periodic basis to allow the playback device tofully refresh a 360 degree version of said scene at least once duringeach refresh period.

What is claimed is:
 1. An image capture system, comprising: a firstcamera, the first camera including: a first sensor; and a first fish eyelens that is asymmetric in at least one of a vertical direction and ahorizontal direction, said first fish eye lens having a shape thatdirects light of a first portion of an environment to a first set ofpixel element sensors per unit area and light of a second portion of theenvironment to a second set of pixel element sensors per unit area, thefirst portion of the environment being above or below the second portionof the environment, said sensor capturing the first portion of theenvironment at a lower resolution than said second potion of theenvironment.
 2. The image capture system of claim 1, wherein said firstfish eye lens is asymmetric in said vertical dimension and symmetric insaid horizontal dimension.
 3. The image capture system of claim 1,wherein said first fish eye lens is asymmetric in both the vertical andhorizontal dimensions.
 4. The image capture system of claim 1, whereinthe first portion of the environment is a sky portion.
 5. The imagecapture system of claim 1, wherein the first fish eye lens directs lightpassing through an upper portion of the first lens onto a first portionof said first sensor and directs light passing through a second portionof the first lens onto a second portion of the first sensor, said secondportion of the first sensor being larger than said first portion.
 6. Theimage capture system of claim 4, wherein said first fish eye lens has ashape that directs light of a third portion of the environment to athird set of pixel element sensors per unit area, the first and secondportions of the environment being above the third portion of theenvironment, sad sensor capturing the third portion of the environmentat a lower resolution than said first and second potions of theenvironment.
 7. The image capture system of claim 1, wherein the firstfish eye lens has a field of view of at least 160 degrees in thevertical direction.
 8. The image capture system of claim 7, wherein thefirst fish eye lens also has a field of view of at least 160 degrees inthe horizontal direction.
 9. The image capture system of claim 1,wherein said first fish eye lens has a center which is offset from acenter of the first sensor in the horizontal dimension by an amountsufficient to avoid capture of a portion of a lens of an adjacent cameraincluded in a stereoscopic camera pair in which said first camera isincluded.
 10. The image capture system of claim 1, further comprising: asecond camera, the second camera including: a second sensor; and asecond asymmetric fish eye lens having a shape which directs light ofthe first portion of the environment to fewer pixel element sensors perunit area of the first portion of the environment than light of thesecond area of the environment is directed onto thereby resulting incapture of the first portion of the environment at a lower resolutionthan said second portion of the environment by said second sensor. 11.The image capture system of claim 10, further comprising: a secondcamera, the second camera including: a second sensor; and a second fisheye lens.
 12. The image capture system of claim 11, wherein said secondfish eye lens is positioned adjacent the first fish eye lens, a portionof said second fish eye lens obscuring a portion of the field of view ofthe first fish eye lens.
 13. The image capture system of claim 12,wherein the first sensor is shifted with respect to the center of thefirst fish eye lens resulting in a portion of the light passing throughthe first fish eye lens missing said first sensor.
 14. The image capturesystem of claim 13, wherein said portion of the light passing throughthe first fish eye lens missing said first sensor including lightcorresponding to the portion of the second fish eye lens that obscuresthe portion of the field of view of the first fish eye lens.
 15. Theimage capture system of claim 13, wherein the second sensor is shiftedwith respect to the center of the second fish eye lens resulting aportion of the light passing through the second fish eye lens missingsaid first sensor, said shift of the second sensor with respect to thecenter of the second fish eye lens being a shift in a horizontaldirection which is equal and opposite the amount the first sensor isshifted with respect to the center of the first fish eye lens.
 16. Theimage capture system of claim 6, further comprising: a second camerapositioned adjacent said first camera.
 17. The image capture system ofclaim 16, wherein said first and second cameras form said stereoscopiccamera pair, the second camera including: a second asymmetric fish eyelens; and wherein said second asymmetric fish eye lens has a centerwhich is offset from a center of a second sensor in the horizontaldimension by an amount sufficient to avoid capture of a portion of thefirst fish eye lens of the first camera included in the stereoscopiccamera pair.